Evaluating and Improving Risk Analysis Methods for Critical Systems

At the same time as our dependence on IT systems increases, the number of reports of problems caused by failures of critical IT systems has also increased. Today, almost every societal system or service, e.g., water supply, power supply, transportation, depends on IT systems, and failures of these systems have serious and negative effects on society. In general, public organizations are responsible for delivering these services to society. Risk analysis is an important activity for the development and operation of critical IT systems, but the increased complexity and size of critical systems put additional requirements on the effectiveness of risk analysis methods. Even if a number of methods for risk analysis of technical systems exist, the failure behavior of information systems is typically very different from mechanical systems. Therefore, risk analysis of IT systems requires different risk analysis techniques, or at least adaptations of traditional approaches. The research objective of this thesis is to improve the analysis process of risks pertaining to critical IT systems, which is addressed in the following three ways. First, by understanding current literature and practices related to risk analysis of IT systems, then by evaluating and comparing existing risk analysis methods, and by suggesting improvements in the risk analysis process and by developing new effective and efficient risk analysis methods to analyze IT systems. To understand current risk analysis methods and practices we carried out a systematic mapping study. The study found only few empirical research papers on the evaluation of existing risk analysis methods. The results of the study suggest to empirically investigate risk analysis methods for analyzing IT systems to conclude which methods are more effective than others. Then, we carried out a semi-structured interview study to investigate several factors regarding current practices and existing challenges of risk analysis and management, e.g., its importance, identification of critical resources, involvement of different stakeholders, used methods, and follow-up analysis. To evaluate and compare the effectiveness of risk analysis methods we carried out a controlled experiment. In that study, we evaluated the effectiveness of risk analysis methods by counting the number of relevant and non-relevant risks identified by the experiment participants. The difficulty level of risk analysis methods and the experiment participants’ confidence about the identified risks were also investigated. Then, we carried out a case study to evaluate the effectiveness and efficiency of existing risk analysis methods, Failure Mode and Effect Analysis (FMEA) and System Theoretic Process Analysis (STPA). The case study investigates the effectiveness of the methods by performing a comparison of how a hazard analysis is conducted for the same system. It also evaluates the analysis process of risk analysis methods by using a set of qualitative criteria, derived from the Technology Acceptance Model (TAM). After this, another case study was carried out to evaluate and assess the resilience of critical IT systems and networks by applying a simulation method. A hybrid modeling approach was used which considers the technical network, represented using graph theory, as well as the repair system, represented by a queuing model. To improve the risk analysis process, this thesis also presents a new risk analysis method, Perspective Based Risk Analysis (PBRA), that uses different perspectives while analyzing IT systems. A perspective is a point of view or a specific role adopted by risk analyst while doing risk analysis, i.e., system engineer, system tester, or system user. Based on the findings, we conclude that the use of different perspectives improves effectiveness of risk analysis process. Then, to improve the risk analysis process we carried out a data mining study to save historical information about IT incidents to be used later for risk analysis. It could be an important aid in the process of building a database of occurred IT incidents that later can be used as an input to improve the risk analysis process. Finally, based on the findings of the studies included in this thesis a list of suggestions is presented to improve the risk analysis process. This list of potential suggestions was evaluated in a focus group meeting. The suggestions are for example, risk analysis awareness and education, defining clear roles and responsibilities, easy-to-use and adapt risk analysis methods, dealing with subjectivity, carry out risk analysis as early as possible and finally using historical risk data to improve the risk analysis process. Based on the findings it can be concluded that these suggestions are important and useful for risk practitioners to improve the risk analysis process.The presented research work in this thesis provides research about methods to improve the risk analysis and management practices. Moreover, the presented work in this thesis is based on solid empirical studies

Hazard Analysis of Collision Avoidance System using STPA

As our society becomes more and more dependent on IT systems, failures of these systems can harm more and more people and organizations both public and private. Diligently performing risk and hazard analysis helps to minimize the societal harms of IT system failures. In this paper we present experiences gained by applying the System Theoretic Process Analysis (STPA) method for hazard analysis on a forward collision avoidance system. Our main objectives are to investigate effectiveness in terms of the number and quality of identified hazards, and time efficiency in terms of required efforts of the studied method. Based on the findings of this study STPA has proved to be an effective and efficient hazard analysis method for assessing the safety of a safety-critical system and it requires a moderate level of effort

Development of Safety-Critical Software Systems Using Open Source Software - A Systematic Map

The popularity of Open Source Software (OSS) has increased the interest in using it in safety critical applications. The aim of this study is to review research carried out on usage of open source code in development of safety-critical software and systems. We conducted a systematic mapping study through searches in library databases and manual identification of articles from open source conferences.We have identified 22 studies about using open source software, mainly in automotive, aerospace, medical and nuclear domains. Moreover, only a few studies present complete safety systems that are released as OSS in full. The most commonly used OSS functionalities are operating systems, imaging, control and data management. Finally most of the integrated OSS have mature code bases and a commit history of more than five years

An Analysis and Comparison of The Security Features of Firewalls and IDSs

In last few years we have observed a significant increase in the usage of computing devices and their capabilities to communicate with each other. With the increase in usage and communicating capabilities the higher level of network security is also required. Today the main devices used for the network security are the firewalls and IDS/IPS that provide perimeter defense. Both devices provide many overlapping security features but they have different aims, different protection potential and need to be used together. A firewall is an active device that implements ACLs and restricts unauthorized access to protected resources. An IDS only provides information for further necessary actions, not necessarily perimeter related, but some of these needed actions can be automated, such as automatic blocking in the firewall of attacking sites, which creates an IPS. This thesis report analyzed some common firewall and IDS products, and described their security features, functionalities, and limitations in detail. It also contains the comparison of the security features of the both devices. The firewall and IDS perform different functions for the network security, so they should be used in layered defense architecture. The passwords, firewalls, IDSs/IPSs and physical security all together provide a layered defense and complement each other. The firewall and IDS alone cannot offer sufficient network protection against the network attacks, and they should be used together to enhance the defense-in-depth or layered approach

Improving Risk Analysis Practices in Governmental Organizations

At the same time as our dependence on IT systems increases, the number of reports of problems caused by failures of critical IT systems has also increased. Today almost every system or service, e.g., water, power supply, transportation, is dependent on IT systems, and failure of these systems has serious and negative effects on society. In general, governmental organizations are responsible for delivery of these services to society. The increasing dependence on critical IT systems also makes them more and more complex. Risk analysis is an important activity for the development and operation of critical IT systems, but the increased complexity and size put additional requirements on the effectiveness of risk analysis methods. Risk analysis of technical systems has a long history in mechanical and electrical engineering. Even if a number of methods for risk analysis of technical systems exist, the failure behavior of information systems is typically very different from mechanical systems. Therefore, risk analysis of IT systems requires different risk analysis techniques, or at least adaptations of traditional approaches. The research objective of this thesis is to improve the analysis process of risks pertaining to IT systems in governmental organizations. In this thesis the improvements in risk analysis processes are addressed in two different ways. First, by understanding what types of methods are available for IT systems and how they can be improved. Second, by developing new effective and efficient risk analysis methods that can be useful to analyze IT systems in governmental organizations. In this thesis work, a systematic mapping study was carried out to understand existing methods and techniques used for analyzing IT systems. It found very few empirical research papers about the evaluation of existing risk analysis methods. The results of the mapping study suggest to empirically investigate risk analysis methods for analyzing IT systems to conclude which methods are more effective than others. Based on the results of the mapping study a case study was carried out to evaluate the effectiveness and efficiency of an existing risk analysis method, System Theoretic Process Analysis (STPA). Based on the results of the mapping study a controlled experiment was carried out to evaluate the effectiveness of risk analysis methods. The effectiveness of risk analysis methods was evaluated by counting the number of relevant and non-relevant risks identified by the experiment participants. The difficulty level of risk analysis methods and the experiment participants’ confidence about the identified risks were also investigated. The work presented in this thesis also presents a new risk analysis method, Perspective Based Risk Analysis (PBRA), that uses different perspectives while analyzing IT systems. A perspective is a point of view or a specific role adopted by risk analyst while doing risk analysis, i.e., system engineer, system tester, or system user. A case study was carried out to save historical information about IT incidents to be used later for risk analysis. This study investigates how difficult it is to find relevant risks from the available sources and the effort required to set up such a system. It also investigates how accurate the found risks are. It is believed that this could be an important aid in the process of building a database of occurred IT incidents that later can be used as an input to improve the risk analysis process. The presented research work in this thesis provides research about methods and tools for governmental organizations to improve their risk analysis and management practices. Moreover, the presented work in this thesis is based on solid empirical studies

Risk Analysis and Management of IT Systems: Practice and Challenges

Risk analysis is important for safety-critical IT systems and services, both in public and private organizations. However, the actual practices and the challenges of risk analysis in these contexts have not been fully explored. This paper investigates the current practices of risk analysis by an interview-based investigation. This study investigates several factors of the risk analysis process, e.g., its importance, identification of critical resources, definitions of roles, involvement of different stakeholders, used methods, and follow-up analysis. Further more, this study also investigates existing challenges in the current practices of risk analysis. A number of challenges are identified,e.g., that risk analysis requires competence both about the risk analysis procedures and the analyzed system,which is challenging to identify, and that it is challenging to follow-up and repeat a risk-analysis that is conducted. The identified challenges can be useful when new risk analysis methods are defined

Identification of IT Incidents for Improved Risk Analysis by Using Machine Learning

Today almost every system or service, e.g., water, power supply, transportation, etc. is dependent on IT systems, and failure of these systems have serious and negative effects on the society. IT incidents are critical for the society as they can stop the function of critical systems and services. Moreover, in a software engineering context risk analysis is an important activity for the development and operation of safe software-intensive systems. However, the increased complexity and size of software intensive systems put additional requirements on the effectiveness of the risk analysis process. Therefore, the risk analysis process needs to be improved and it is believed that by having an overview of already occurred IT incidents, the risk analysis process can be improved. The saved information about IT incidents can be used as an input to risk analysis, which can help to correctly estimate the consequences of potential risks. This study investigates how difficult is it to find relevant risks from the available sources and the effort required to set up such a system. It also investigates how accurate are the found risks. It presents a prototype solution of a system that automatically identifies information pertaining to IT incidents, from texts available online on Internet news sources, that have happened. This way IT incidents can be saved semi-automatically in a database and the saved information can be used later as an input to risk analysis. In this study 58% of texts that potentially can contain information about IT incidents were correctly identified from an experiment dataset by using the presented method. It is concluded that the identifying texts about IT incidents with automated methods like the one presented in this study is possible, but it requires some effort to set up

An Industrial Case Study on Measuring the Quality of the Requirements Scoping Process

Decision making and requirements scoping occupy central roles in helping to develop products that are demanded by the customers and ensuring company strategies are accurately realized in product scope. Many companies experience continuous and frequent scope changes and fluctuations but struggle to measure the phenomena and correlate the measurement to the quality of the requirements process. We present the results from an exploratory interview study among 22 participants working with requirements management processes at a large company that develops embedded systems for a global market. Our respondents shared their opinions about the current set of requirements management process metrics as well as what additional metrics they envisioned as useful. We present a set of metrics that describe the quality of the requirements scoping process. The findings provide practical insights that can be used as input when introducing new measurement programs for requirements management and decision making